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ABSTRACT 



This report describes one of a series of pilot studies 
conducted to evaluate the instructional effectiveness of the Harcourt Reading 
Program. The study focused on the program’s primary levels and included 
Grades 1 and 3. Participants, teachers who had not used the program 
previously, were encouraged to select one theme to pilot over a 6 -week 
period; they also agreed to administer data collection instruments before and 
after instruction. A total of 33 teachers from 15 different schools in 7 
different states volunteered to participate- -21 at Grade 1 and 12 at Grade 3. 
A quasi-experimental pretest-posttest design was used. Before instruction 
students were administered two pretests; following the pretests, teachers 
taught the selected book or theme using resources and procedures contained in 
the Harcourt Reading Program. Students were administered the post tests upon 
completion of the selected theme. Results indicated: (1) increase in test 

scores on both of the national standardized tests, the Stanford Achievement 
Tests and the Metropolitan Achievement Tests, and on the Harcourt Reading 
Program assessment are both positive and statistically significant for all 
subtests and total test scores at both grades; (2) gains on both the 
instructional assessments and the standardized tests were approximately the 
same, indicating the results were not due to the focus of the instructional 
assessments; (3) gains of this magnitude are remarkable given that teachers 
volunteered to teach the units and did not receive extra training; (4) 
increases were greater for Grade 1 than for Grade 3; and (5) it is 
significant that no test scores stayed the same--they all increased. Contains 
11 tables of data. (NKA) 
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Instructional Effectiveness of the Harcourt Reading Program 



A Study of the Instructional Effectiveness 
of the HARCOURT READING Program 



Jennifer M. Conner 
Beth Greene 

Educational Research Institute of America 

This report describes one of a series of pilot studies that was conducted to evaluate the 
instructional effectiveness of the HARCOURT READING Program. 

Background Information 

Harcourt School Publishers contracted with the Educational Research Institute of America 
(ERIA) to conduct a series of independent pilot studies to evaluate the effectiveness of the 
HARCOURT READING Program. Harcourt School Publishers sought out volunteer teachers to 
participate in the study; the Harcourt Department of Test Services scored the national 
standardized achievement tests; and ERIA conducted the study, scored the Harcourt Skills 
Assessments and analyzed all of the data that were collected. The study described in this report 
was conducted in the spring of the 1999-2000 school year. 

Research Questions 

The following research questions guided the design, procedures, and data analysis of the study: 

Is the HARCOURT READING Program instructionally effective? Do selected themes 
significantly increase students’ understanding of key reading skills, concepts, and 
strategies as measured by the program’s Reading Skills Assessments? As measured by 
standardized achievement tests (Stanford 9 or Metropolitan Achievement Test)? 

Design and Procedures of the Study 

This study focused on the primary levels of the program and included Grades 1 and 3. Other 
ERIA studies conducted under contract to Harcourt School Publishers included other primary 
grades. 

Teachers who volunteered to participate in the study had not used the program previously. The 
teachers were encouraged to select one theme to pilot over a six-week period. They also agreed 
to administer data collection instruments before beginning instruction and again after completing 
instruction. 

A total of 33 teachers volunteered to participate in the study: 21 at Grade 1 and 12 at Grade 3. 
The participating teachers came from 15 different schools in 7 different states. 
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A quasi-experimental pretest-posttest design was used. Before instruction began, students were 
administered two pretests. The classroom teachers administered all tests. Table 1 summarizes 
the data collection instruments that were used. 

TABLE 1 

Data Collection Instruments 



Grade 


Pretests 


Posttests 


1 


• Harcourt Reading Skills Assessments 
(matched to each book taught) 

• Reading Subtests of either the Stanford 
Achievement Test, Ninth Edition 
(Abbreviated Battery), Level Pl/Form S 
or the Metropolitan Achievement Test, 
Seventh Edition, Level Pl/Form S 


• Harcourt Reading Skills Assessments 
(matched to each book taught) 

• Reading Subtests of either the Stanford 
Achievement Test, Ninth Edition 
(Abbreviated Battery), Level Pl/Form S 
or the Metropolitan Achievement Test, 
Seventh Edition, Level Pl/Form S 


3 


• Harcourt Reading Skills Tests (matched 
to each theme taught) 

• Reading Subtest of either the Stanford 
Achievement Test, Ninth Edition 
(Abbreviated Battery), Level P3/Form S 
or the Metropolitan Achievement Test, 
Level ELEM 1/Form S 


• Harcourt Reading Skills Tests (matched 
to each theme taught) 

• Reading Subtest of either the Stanford 
Achievement Test, Ninth Edition 
(Abbreviated Battery), Level P3/Form S 
or the Metropolitan Achievement Test, 
Level ELEMl/Form S 



Following the pretests, the teachers taught the selected book or theme using the resources and 
procedures contained in the Harcourt Reading Program. Eighteen of the Grade 1 teachers chose 
to use Book 1- 4, Welcome Home, while only three of the Grade 1 teachers chose to use Book 1- 
5, Set Sail. Grade 3 teachers used a total of five different themes selected from either Hidden 
Surprises or Journeys of Wonder. While each theme took 6 weeks to teach, most teachers started 
the pretests in February and completed the posttests in May. Instruction included a broad sample 
of reading skills from the strands of decoding, vocabulary, comprehension, literary appreciation, 
and study skills. 

Upon completion of the selected theme, students were administered the posttests. All data 
collection instruments were returned to the Educational Research Institute of America where 
they were processed. The Stanford 9 Achievement Tests and the Metropolitan Achievement Tests 
were scored at the Harcourt Educational Measurement Scoring Center. The Harcourt Skills 
Assessments were scored at ERIA, and all of the data were analyzed by ERIA. 
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Findings 

Descriptions of each of the assessments used in the study and the assessment results are reported 
for Grades 1 and 3 in this section of the report. 

TABLE 2 



Content of the Grade 1 Reading Skills Assessment Themes Welcome Home and SetSail 



Grade 1 Reading Skills Assessment 


Theme: 

Welcome Home 


Theme: 
Set Sail 


Subtest 


No. of Items 


No. of Items 


Decoding 






Diagraphs-Sound Letter Relationships 


4 




Vowels-Decodable Words 


6 


4 


Word Structure-Inflections 




4 


Word Structure-Contractions 


4 


4 


Vocabulary 


16 


20 


Comprehension 






Main Idea 




4 


Noting Details 


4 




Make Predictions 


4 




Study Skills 






Alphabetical Order 


4 




Language 






Pronouns 


4 




Describing Words 


4 




Verbs 




4 


TOTAL 


50 


40 



m&drn 




§1 

Si 









Table 3 summarizes the Pretest and Posttest means and standard deviations for the Grade 1 
Reading Skills Assessment for Welcome Home. 

TABLE 3 

Grade 1 Theme: Welcome Home Results (N-311) 



Grade 1 


Reading Skills Assessment - Book 1-4 (50 items) 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest 


8 


50 


41.03 (82%) 


7.80 


Posttest 


24 


50 


45.87 (91%) 


5.02 



A paired t-test for the mean Reading Skills Assessment scores showed that the scores improved 
significantly after instruction (t=14.625; p < .0001). 
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TABLE 4 

Grade 1 Theme: Set Sail Results (N=54) 



Grade 1 


Reading Skills Assessment - Book 5 (40 items) 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest 


26 


39 


34.75 (86%) 


3.49 


Posttest 


28 


39 


36.19(91%) 


3.01 



A paired t-test for the mean Reading Skills Assessment scores showed that the scores improved 
significantly after instruction (t=4.04; p < .0001). 

TABLE 5 
Grade 1 Results 



Grade 1 


Stanford Achievement Tests: Grade 1 (N=117) 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest: Word 
Study Skills 


4 


20 


13.06 (65%) 


3.63 


Posttest: Word 
Study Skills 


8 


20 


15.32 (77%) 


3.37 


Pretest: Word 
Reading 


2 


20 


12.73 (64%) 


4.64 


Posttest: Word 
Reading 


3 


20 


14.87 (74%) 


4.01 


Pretest: Reading 
Comprehension 


7 


30 


19.38 (65%) 


5.86 


Posttest: Reading 
Comprehension 


10 


30 


23.00 (77%) 


5.35 


Pretest: Total 
Reading 


20 


68 


45.33 (65%) 


12.67 


Posttest: Total 
Reading 


24 


69 


53.78 (77%) 


11.27 



A paired t-test for the four subtests and the total test score on the Stanford Achievement Tests: 
Grade One resulted in a significant increase (Word Study Skills: 1=7.82; pc.OOOl); (Word 
Reading: 1=5.57; p<.0001); (Reading Comprehension: 1=10.45; pc.OOOl); (Total Reading: 
1=11.40; pc.0001). 
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TABLE 6 
Grade 1 Results 



Grade 1 


Metropolitan 


Achievement Tests: Grade ‘\(N=225) 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest: Word 
Recognition 


6 


30 


17.86 (59%) 


5.03 


Posttest: Word 
Recognition 


7 


30 


21.15 (70%) 


5.37 


Pretest: 

Vocabulary 


2 


24 


12.20 (51%) 


6.06 


Posttest: 

Vocabulary 


2 


24 


15.75 (66%) 


5.77 


Pretest: Reading 
Comprehension 


4 


37 


17.94 (45%) 


7.51 


Posttest: Reading 
Comprehension 


8 


39 


22.59 (57%) 


8.25 


Pretest: Total 
Reading 


21 


90 


48.55 (52%) 


16.45 


Posttest: Total 
Reading 


24 


91 


59.58 (63%) 


17.24 



A paired t-test for the three subtests and the total test score on the Stanford Achievement Tests: 
Grade One resulted in a significant increase (Word Recognition: r=10.610; pc.OOOl); (Word 
Vocabulary: r= 1 3 .64 1 ; pc.OOOl); (Reading Comprehension: /=12.318; pc.OOOl); (Total Reading: 
/=18.290; pc.OOOl). 
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Table 7 

Grade 3 Reading Skills Results 



Grade 3 Reading Skills Assessment 

Subtests 


Journeys 
Theme 1 


Journeys 
Theme 2 


Journeys 
Theme 3 


Hidden 
Surprises 
Theme 1 


Hidden 
Surprises 
Theme 2 


Decoding 












Prefixes and Suffixes 








6 




Vocabulary 












Selection Vocabulary 


18 


14 


14 


20 


16 


Vocabulary in Context 










6 


Comprehension 












Sequence 


4 










Fact and Opinion 


8 










Compare and Contrast 




4 








Summarize 




4 








Important Details 






8 






Cause and Effect 






8 






Predict Outcomes 








4 




Draw Conclusions 










4 


Literary Concepts 












Character’s Feelings and Actions 




4 








Story Elements 










4 


Study and Research Skills 












Graphic Sources 




4 








Total 


30 


30 


30 


30 


30 



Table 8 summarizes the means and standard deviations of the scores on the pretest and the 
posttest for the Grade 3 Reading Skills Assessments. 



TABLE 8 

Grade 3 Results: Reading Skills Assessment ( N=2H ) 



Grade Three 


Reading Skills Assessment (30 items) 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest 


5 


30 


23.14 (77%) 


7.58 


Posttest 


8 


30 


25.01 (84%) 


5.47 



A paired t-test for the mean Reading Skills Assessment scores showed that the scores improved 
significantly after instruction (t=3.948; p < .0001). 
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TABLE 9 



Grade 3 


Stanford Achievement Tests: Grad 


i 

CO 

V 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest: Reading 
Vocabulary 


5 


19 


12.85 (64%) 


3.77 


Posttest: Reading 
Vocabulary 


5 


20 


13.93 (70%) 


3.72 


Pretest: Reading 
Comprehension 


2 


30 


16.39 (55%) 


5.94 


Posttest: Reading 
Comprehension 


2 


30 


18.16 (61%) 


6.01 


Pretest: Total 
Reading 


11 


49 


29.04 (58%) 


8.29 


Posttest: Total 
Reading 


15 


49 


32.54 (65%) 


8.64 



A paired t-test for the three subtests and the total test score on the Stanford Achievement Tests: 
Grade Three resulted in a significant increase (Reading Vocabulary: r=2.95; p<.005); (Reading 
Comprehension: r=3.21; p<.002); (Total Reading: t= 3.73; pc.0001). 



TABLE 10 



Grade 3 


Metropolitan Achievement Tests: Gr; 


ade 3 (N=87) 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest: Reading 
Vocabulary 


2 


27 


14.14 (47%) 


6.02 


Posttest: Reading 
Vocabulary 


5 


28 


17.88 (60%) 


6.10 


Pretest: Reading 
Comprehension 


6 


49 


29.21 (53%) 


11.44 


Posttest: Reading 
Comprehension 


6 


51 


31.24 (57%) 


11.65 


Pretest: Total 
Reading 


13 


74 


43.40 (51%) 


16.27 


Posttest: Total 
Reading 


11 


78 


49.27 (58%) 


17.02 



A paired t-test for the three subtests and the total test score on the Metropolitan Achievement 
Tests: Grade Three resulted in a significant increase (Reading Vocabulary: /=8.915; pc.OOOl); 
(Reading Comprehension: f=3.173; pc.OOOl); (Total Reading: t=7.138; pc.OOOl). 
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Summary of Results 

• The increase in test scores on both of the nationally standardized tests, the Stanford 
Achievement Tests and the Metropolitan Achievement Tests , and on the Harcourt Reading 
Program assessment are both positive and statistically significant for all subtests and total test 
scores at both Grades 1 and 3. 

• The gains on both the instructional assessments and the nationally standardized tests were 
approximately the same indicating the results were not due to the focus of the instructional 
assessments. The same increases were found on the nationally standardized tests that cover a 
wider range of skills than the Harcourt Instructional Assessments. 

• Gains of this magnitude for such a brief period of instruction are quite remarkable 
considering that the teachers volunteered to teach the units and did not receive any extra 
training. 

• Some of the results may have shown even greater gains. However, there was a ceiling effect 
for some of the assessments. (Students scored perfect or almost perfect scores on the posttests 
thus limiting the gain scores.) 

• The increases were greater for Grade 1 than for Grade 3. This result suggests (as do many 
studies of this sort) that getting programs established in the earliest grades provides the 
greatest possibility of success. 

• It is also significant that no test scores stayed the same. They all increased. In a short-term 
study of this sort it is not uncommon to note some test scores that do not increase at all. 

The percent of gains on each of the tests is shown in Table 11: 

TABLE 11 

Summary of Test Score Increases 





Percent Correct 
on Pretest 


Percent Correct 
on Posttest 


Percent 
Gain from 
Pretest to 
Posttest 


Grade/Test 




Grade 1: Theme 1 Assessment 


82% 


91% 


+9% 


Grade 1: Theme 1 Assessment 


86% 


91% 


+5% 


Stanford Achievement Test 
Grade 1: Total Score 


65% 


77% 


+12% 


Metropolitan Achievement Test 
Grade 1: Total Score 


52% 


63% 


+11% 


Grade 3: Themed Assessments 


77% 


84% 


+7% 


Stanford Achievement Test 
Grade 3: Total Score 


58% 


65% 


+7% 


Averages 


70% 


79% 


+9% 
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